ir_metadata: An Extensible Metadata Schema for IR Experiments

The information retrieval (IR) community has a strong tradition of making the computational artifacts and resources available for future reuse, allowing the validation of experimental results. Besides the actual test collections, the underlying run files are often hosted in data archives as part of conferences like TREC, CLEF, or NTCIR. Unfortunately, the run data itself does not provide much information about the underlying experiment. For instance, the single run file is not of much use without the context of the shared task's website or the run data archive. In other domains, like the social sciences, it is good practice to annotate research data with metadata. In this work, we introduce ir_metadata - an extensible metadata schema for TREC run files based on the PRIMAD model. We propose to align the metadata annotations to PRIMAD, which considers components of computational experiments that can affect reproducibility. Furthermore, we outline important components and information that should be reported in the metadata and give evidence from the literature. To demonstrate the usefulness of these metadata annotations, we implement new features in repro_eval that support the outlined metadata schema for the use case of reproducibility studies. Additionally, we curate a dataset with run files derived from experiments with different instantiations of PRIMAD components and annotate these with the corresponding metadata. In the experiments, we cover reproducibility experiments that are identified by the metadata and classified by PRIMAD. With this work, we enable IR researchers to annotate TREC run files and improve the reuse value of experimental artifacts even further.Comment: Resource pape

Simulating Users in Interactive Web Table Retrieval

Considering the multimodal signals of search items is beneficial for retrieval effectiveness. Especially in web table retrieval (WTR) experiments, accounting for multimodal properties of tables boosts effectiveness. However, it still remains an open question how the single modalities affect user experience in particular. Previous work analyzed WTR performance in ad-hoc retrieval benchmarks, which neglects interactive search behavior and limits the conclusion about the implications for real-world user environments. To this end, this work presents an in-depth evaluation of simulated interactive WTR search sessions as a more cost-efficient and reproducible alternative to real user studies. As a first of its kind, we introduce interactive query reformulation strategies based on Doc2Query, incorporating cognitive states of simulated user knowledge. Our evaluations include two perspectives on user effectiveness by considering different cost paradigms, namely query-wise and time-oriented measures of effort. Our multi-perspective evaluation scheme reveals new insights about query strategies, the impact of modalities, and different user types in simulated WTR search sessions.Comment: 4 pages + references; accepted at CIKM'2

Evaluating Temporal Persistence Using Replicability Measures

In real-world Information Retrieval (IR) experiments, the Evaluation Environment (EE) is exposed to constant change. Documents are added, removed, or updated, and the information need and the search behavior of users is evolving. Simultaneously, IR systems are expected to retain a consistent quality. The LongEval Lab seeks to investigate the longitudinal persistence of IR systems, and in this work, we describe our participation. We submitted runs of five advanced retrieval systems, namely a Reciprocal Rank Fusion (RRF) approach, ColBERT, monoT5, Doc2Query, and E5, to both sub-tasks. Further, we cast the longitudinal evaluation as a replicability study to better understand the temporal change observed. As a result, we quantify the persistence of the submitted runs and see great potential in this evaluation method.Comment: To be published in Proceedings of the Working Notes of CLEF 2023 - Conference and Labs of the Evaluation Forum, Thessaloniki, Greece 18 - 21, 202

A Living Lab Architecture for Reproducible Shared Task Experimentation

No existing evaluation infrastructure for shared tasks currently supports both reproducible on- and offline experiments. In this work, we present an architecture that ties together both types of experiments with a focus on reproducibility. The readers are provided with a technical description of the infrastructure and details of how to contribute their own experiments to upcoming evaluation tasks

Integration of the 3D Environment for UAV Onboard Visual Object Tracking

Single visual object tracking from an unmanned aerial vehicle (UAV) poses fundamental challenges such as object occlusion, small-scale objects, background clutter, and abrupt camera motion. To tackle these difficulties, we propose to integrate the 3D structure of the observed scene into a detection-by-tracking algorithm. We introduce a pipeline that combines a model-free visual object tracker, a sparse 3D reconstruction, and a state estimator. The 3D reconstruction of the scene is computed with an image-based Structure-from-Motion (SfM) component that enables us to leverage a state estimator in the corresponding 3D scene during tracking. By representing the position of the target in 3D space rather than in image space, we stabilize the tracking during ego-motion and improve the handling of occlusions, background clutter, and small-scale objects. We evaluated our approach on prototypical image sequences, captured from a UAV with low-altitude oblique views. For this purpose, we adapted an existing dataset for visual object tracking and reconstructed the observed scene in 3D. The experimental results demonstrate that the proposed approach outperforms methods using plain visual cues as well as approaches leveraging image-space-based state estimations. We believe that our approach can be beneficial for traffic monitoring, video surveillance, and navigation.Comment: Accepted in MDPI Journal of Applied Science

The role of the information environment during the first COVID-19 wave in Germany

The COVID-19 pandemic has been accompanied by intense debates about the role of the information environment. On the one hand, citizens learn from public information campaigns and news coverage and supposedly adjust their behaviours accordingly; on the other, there are fears of widespread misinformation and its detrimental effects. Analyzing the posts of the most important German information providers published via Facebook, this paper first identifies a uniform salience of subtopics related to COVID-19 across different types of information sources that generally emphasized the threats to public health. Next, using a large survey conducted with German residents during the first COVID-19 wave in March 2020 we investigate how information exposure relates to perceptions, attitudes and behaviours concerning the pandemic. Regression analyses show that getting COVID-19-related information from a multitude of sources has a statistically significant and positive relationship with public health outcomes. These findings are consistent even across the ideological left/right spectrum and party preferences. These consistent correlational results demonstrate that during the first wave of COVID-19, a uniform information environment went hand in hand with a cautious public and widely accepted mitigation measures. Nonetheless, we discuss these findings against the backdrop of an increased politicization of public-health measures during later COVID-19 waves

ifo Konjunkturprognose 2009/2010: Abschwung setzt sich fort

Die Weltwirtschaft steckt in der tiefsten Rezession seit der großen Depression. Seit diesem Frühjahr dürfte sich das Kontraktionstempo jedoch abgeschwächt haben. Weltweit sind Programme zur Stützung der Konjunktur aufgelegt worden, zudem zeigt die expansive Geldpolitik der Zentralbanken allmählich Wirkung. Auch sind in zahlreichen Ländern Maßnahmen zur Stabilisierung der Finanzmärkte ergriffen worden. Schließlich werden die Realeinkommen durch die starken Rohstoffpreisrückgänge gestützt. Das vom ifo Institut erhobene Weltwirtschaftsklima ist im zweiten Quartal 2009 erstmals seit dem Herbst 2007 wieder gestiegen. Der Anstieg des Indikators resultiert jedoch ausschließlich aus den günstigeren Erwartungen für die nächsten sechs Monate; die Einschätzung der derzeitigen wirtschaftlichen Lage hat sich demgegenüber noch weiter verschlechtert und fiel auf einen neuen historischen Tiefstand. Die deutsche Wirtschaft befindet sich in der schwersten Rezession seit Bestehen der Bundesrepublik. Nach den bisher vorliegenden amtlichen Ergebnissen sank die gesamtwirtschaftliche Produktion im ersten Quartal 2009 saison- und kalenderbereinigt gegenüber dem Jahresendquartal um 3,8%, bereits im vierten Quartal 2008 war die Wirtschaftsleistung um 2,2% geschrumpft. Deutschland verzeichnet damit von allen großen europäischen Volkswirtschaften den schärfsten Wachstumseinbruch. Die gesamtwirtschaftliche Produktion dürfte nach diesem drastischen Rückgang auch im zweiten Quartal gesunken sein, jedoch mit deutlich verringertem Tempo. Für den Beginn einer allmählichen Stabilisierung spricht die aktuelle Entwicklung einer Reihe von gewichtigen Konjunkturindikatoren wie Produktion und Auftragseingang sowie das ifo Geschäftsklima. Zum Rückgang der Wirtschaftsleistung tragen vor allem die Industrie und der Sektor Finanzierung, Vermietung und Unternehmensdienstleistungen bei.Konjunktur, Konjunkturumfrage, Konjunkturprognose, Wirtschaftslage, Geschäftsklima, Weltkonjunktur, Deutschland, Welt

ifo Konjunkturprognose 2010: Deutsche Wirtschaft ohne Dynamik

Am 15. Dezember 2009 stellte das ifo Institut seine Prognose für die Jahre 2010 und 2011 vor. Zwar steigen seit dem Frühjahr 2009 Produktion und Handel wieder, aber das Wachstumstempo wird niedrig bleiben. Ein Kernproblem ist die Schwächung der internationalen Finanzmärkte, deren Funktionsfähigkeit in wichtigen Segmenten nach wie vor eingeschränkt ist. Zudem musste das Bankensystem gewaltige Eigenkapitalverluste hinnehmen. In wichtigen Industrieländern kommt eine Krise am Immobilienmarkt hinzu, die dort den Abschreibungsbedarf bei Banken erhöht. Die Kreditbedingungen bleiben damit weltweit restriktiv, was die Finanzierung von Investitionen und neuen Arbeitsplätzen massiv beeinträchtigt. Insgesamt wird das Bruttoinlandsprodukt der Welt im Jahr 2010 um 3,1% und im Jahr 2011 um 2,6% steigen, nachdem es im Jahr 2009 um 1,1% zurückgegangen ist. Der Anstieg der Preise wird sich weltweit etwas beschleunigen. In Deutschland hat sich die gesamtwirtschaftliche Produktion im Frühjahr stabilisiert. Im zweiten Quartal expandierte das reale Bruttoinlandsprodukt saison- und kalenderbereinigt um 0,4%, im dritten Quartal sogar um 0,7%. Aufgrund des außerordentlich kräftigen Einbruchs im vorausgegangenen Winterhalbjahr um fast 6% blieb die gesamtwirtschaftliche Produktion, und hier insbesondere die Erzeugung in den exportabhängigen Branchen, aber auf niedrigem Niveau. Der gesamtwirtschaftliche Auslastungsgrad – zieht man die ifo Kapazitätsauslastung im verarbeitenden Gewerbe als Proxy heran – liegt derzeit um etwa 10 Prozentpunkte unter dem langjährigen Durchschnitt. Insgesamt dürfte die gesamtwirtschaftliche Produktion saison- und kalenderbereinigt im Jahresendquartal 2009 weiter zugenommen haben, wenngleich in einem etwas langsameren Tempo (0,5%) als im Vorquartal. Im Gesamtjahr 2009 sinkt das reale Bruttoinlandsprodukt um 4,9%. Im Jahresdurchschnitt 2010 wird das reale Bruttoinlandsprodukt voraussichtlich um 1,7% zunehmen; im Jahresdurchschnitt 2011 um 1,2%.Konjunktur, Konjunkturumfrage, Konjunkturprognose, Wirtschaftslage, Geschäftsklima, Weltkonjunktur, Deutschland, Welt

Teriflunomide treatment for multiple sclerosis modulates T cell mitochondrial respiration with affinity-dependent effects

International audienceInterference with immune cell proliferation represents a successful treatment strategy in T cell-mediated autoimmune diseases such as rheumatoid arthritis and multiple sclerosis (MS). One prominent example is pharmacological inhibition of dihydroorotate dehydrogenase (DHODH), which mediates de novo pyrimidine synthesis in actively proliferating T and B lymphocytes. Within the TERIDYNAMIC clinical study, we observed that the DHODH inhibitor teriflunomide caused selective changes in T cell subset composition and T cell receptor repertoire diversity in patients with relapsing-remitting MS (RRMS). In a preclinical antigen-specific setup, DHODH inhibition preferentially suppressed the proliferation of high-affinity T cells. Mechanistically, DHODH inhibition interferes with oxidative phosphorylation (OXPHOS) and aerobic glycolysis in activated T cells via functional inhibition of complex III of the respiratory chain. The affinity-dependent effects of DHODH inhibition were closely linked to differences in T cell metabolism. High-affinity T cells preferentially use OXPHOS during early activation, which explains their increased susceptibility toward DHODH inhibition. In a mouse model of MS, DHODH inhibitory treatment resulted in preferential inhibition of high-affinity autoreactive T cell clones. Compared to T cells from healthy controls, T cells from patients with RRMS exhibited increased OXPHOS and glycolysis, which were reduced with teriflunomide treatment. Together, these data point to a mechanism of action where DHODH inhibition corrects metabolic disturbances in T cells, which primarily affects profoundly metabolically active high-affinity T cell clones. Hence, DHODH inhibition may promote recovery of an altered T cell receptor repertoire in autoimmunity

Crowdsourcing hypothesis tests: Making transparent how design choices shape research results

To what extent are research results influenced by subjective decisions that scientists make as they design studies? Fifteen research teams independently designed studies to answer fiveoriginal research questions related to moral judgments, negotiations, and implicit cognition. Participants from two separate large samples (total N > 15,000) were then randomly assigned to complete one version of each study. Effect sizes varied dramatically across different sets of materials designed to test the same hypothesis: materials from different teams renderedstatistically significant effects in opposite directions for four out of five hypotheses, with the narrowest range in estimates being d = -0.37 to +0.26. Meta-analysis and a Bayesian perspective on the results revealed overall support for two hypotheses, and a lack of support for three hypotheses. Overall, practically none of the variability in effect sizes was attributable to the skill of the research team in designing materials, while considerable variability was attributable to the hypothesis being tested. In a forecasting survey, predictions of other scientists were significantly correlated with study results, both across and within hypotheses. Crowdsourced testing of research hypotheses helps reveal the true consistency of empirical support for a scientific claim.</div